Document grouped (ColGroup) columns in search downloads - #30
Conversation
Search-result downloads describe each column in a header comment and a definitions section at the bottom of the file. For sections whose columns are grouped with a ColGroup and downloaded as a nested list of subcolumn values (the abstract subgroup search: Subgroup/Ambient/Quotient), the header listed only the group titles (e.g. [Label, Subgroup, Ambient, Quotient]) and the definitions loop skipped the groups entirely (their knowl is None), so the individual subcolumns were never described. Add SearchCol.download_subcols(info) (overridden in ColGroup) that returns a group's subcolumns exactly when it downloads as a nested list (download_col is None and subcols is a concrete list); ColGroups that set download_col (cmf traces/atkin_lehner, cc power_cols) download as a single value and are unchanged. In downloader.py, use it to (a) spell out each grouped column's subcolumn titles under the "Each entry ... has the form" comment and (b) expand each grouped column in the definitions section, documenting every subcolumn by its short title, name and knowl. Verified by generating sage/magma/gp/text downloads for the subgroup search (now documents all 18/2/11 subcolumns matching the data), plus regressions for cmf newforms, conjugacy classes and number fields (byte-identical headers); the downloaded sage/gp/magma files parse and run. cmf + abstract-group + number field download tests pass; pyflakes/ruff clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…MFDB#6477) Addresses review feedback on the grouped (ColGroup) download documentation. The schema of a download existed only as prose passed to lang.comment, which CSVLanguage drops (CSV has no comment syntax), so a CSV subgroup download still had a header of just [label, subgroup_cols, ambient_cols, quotient_cols] over cells holding nested lists. Downloader.make_schema now builds one ColumnSchema per downloaded column (storage name, title, and the ordered subcolumn schemas when the column downloads as a nested list), and that object drives all three places a download file describes itself: the header comment, the definitions section, and assign_columns. CSVLanguage.assign_columns uses it to annotate each grouped header cell with its ordered subcolumn names, keeping one CSV field per top-level column so the data shape and the knowl hyperlinks are unchanged. The first entry of each subgroup group downloads the raw [label, TeX name] pair rather than the displayed link, so sub_name, ambient_name and quotient_name now carry a download_desc spelling out that pair, including when the label is null (group not in the LMFDB) and when both entries are null (quotient of a non-normal subgroup). Verified by generating subgroup downloads in all seven offered formats and checking that number field (ordinary columns) and cmf (ColGroups with a download_col) downloads are byte-identical in all seven formats. Adds four tests, all of which fail against the download machinery as it was before this branch: the text header/definitions/row lengths, the CSV header parsed with the csv module, the scalar ColGroup non-regression, and the download_subcols contract. pyflakes, pylint and ruff clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Thanks for the review. All three points are addressed in the new commit, and the PR description now covers the added tests and the formats verified. 1. CSV is now self-describing. The schema is no longer prose passed to There is still exactly one CSV field per top-level column, so the rows are not flattened; ordinary headers and the knowl hyperlinks are untouched, and groups with a 2. The
with the quotient adding that both entries are null when the subgroup is not normal. Both null cases are real in the data (for example 3. Tests. Four new tests, all of which fail against the download machinery as it was before this branch:
All are bounded to the subgroups of a single group ( Verification.
One judgement call worth flagging: |
|
GPT signed off. |
|
Superseded by LMFDB#7166, opened upstream from this same branch. Closing here; review continues upstream. |
2 participants
Search-result downloads describe each column in a header comment and in a definitions section at the
bottom of the file. For sections whose columns are grouped with a
ColGroupthat downloads as a nestedlist of its subcolumns' values (the abstract subgroup search:
Subgroup/Ambient/Quotient), theheader listed only the group titles (
[Label, Subgroup, Ambient, Quotient]) and the definitions loopskipped the groups entirely (their knowl is
None), so the individual subcolumns were never described.This adds
SearchCol.download_subcols(overridden inColGroup), which identifies exactly the groupsthat download as a nested list, and builds one
ColumnSchemaper downloaded column inDownloader.make_schema. That single schema is then shared by the three places where a download filedescribes its own contents:
lists";
download_desc);assign_columnsnow receives the schema, so the CSV header row, which has no comments to fall backon, annotates each grouped cell with its ordered subcolumn names, for example
ambient_cols [ambient_name, ambient_order]. There is still exactly one CSV field per top-levelcolumn, so the shape of the data is unchanged, and the existing knowl hyperlinks are preserved.
The first entry of each group is a
MultiProcessedColthat downloads the raw[label, TeX name]pairrather than the displayed link, so
sub_name,ambient_nameandquotient_namenow carry adownload_descspelling that pair out, including the cases where the label is null (the group is not inthe LMFDB) and where both entries are null (the quotient of a non-normal subgroup).
ColGroups that setdownload_col(cmf traces/Atkin-Lehner, conjugacy class powers) download as asingle value and are unchanged, as are all non-grouped columns.
New automated coverage:
test_subgroup_search_download_text: the top-level form, the ordered group expansions (first and lastsubcolumn plus representative ones in between, so a truncated or reordered expansion fails), the
grouped-column introductions, the per-subcolumn definitions, the
[label, TeX name]descriptions, andthat each downloaded row matches the documented top-level and per-group lengths.
test_subgroup_search_download_csv: parsed with thecsvmodule. Four header fields, the ordinarylabelcell still just its knowl hyperlink, each grouped cell listing its subcolumns in order, andevery data row with the same field count as the header.
test_scalar_colgroup_download: cmftraces/atkin_lehnerand conjugacy classpower_colskeeptheir
download_col, report no subcolumns and download a single value, while the three subgroupgroups do expand.
test_download_subcolsinlmfdb/tests/test_utils.py: the API contract, namely ordinary column to[], static group withoutdownload_colto its ordered subcolumns, and static or callable group withdownload_colto[].Verified by generating subgroup downloads in all seven offered formats (text, csv, sage, magma, gp, gap,
oscar) and checking that number field (ordinary columns) and cmf (scalar
ColGroups) downloads arebyte-identical in all seven formats to before this change. All four new tests fail against the download
machinery as it was before this branch. pyflakes, pylint (with CI's flags) and ruff are clean.
Addresses LMFDB#6477.
🤖 Generated with Claude Code